EDA Case Study: Titanic

1. Task Description

Titanic is a classical Kaggle competition. The task is to predicts which passengers survived the Titanic shipwreck. For more detail, refer to https://www.kaggle.com/c/titanic/overview/description.

2. Goal of this notebook

As it is a famous competition, there exists lots of excelent analysis on how to do eda and how to build model for this task. See https://www.kaggle.com/startupsci/titanic-data-science-solutions for a reference. In this notebook, we will show how EDAx can simplify the eda process using a few lines of code.

3. Load data

[1]:
from edax import *
import pandas as pd
train_df = pd.read_csv('train.csv')

4. Glimpse of the data

The first thing we need to do is to rounghly understand the data. I.e., how many columns are available, which columns are categorical, which columns are numerical, and which column contains missing values. In EDAx, all of the above questions could be answered in just one line of code!

[2]:
plot(train_df)
[2]:
DataPrep.EDA Report

Dataset Statistics

Number of Variables 12
Number of Rows 891
Missing Cells 866
Missing Cells (%) 8.1%
Duplicate Rows 0
Duplicate Rows (%) 0.0%
Total Size in Memory 315.0 KB
Average Row Size in Memory 362.1 B
Variable Types
  • Numerical: 7
  • Categorical: 5

Dataset Insights

PassengerId is uniformly distributed Uniform
Age has 177 (19.87%) missing values Missing
Cabin has 687 (77.1%) missing values Missing
Survived is skewed Skewed
Pclass is skewed Skewed
SibSp is skewed Skewed
Parch is skewed Skewed
Fare is skewed Skewed
Name has a high cardinality: 891 distinct values High Cardinality
Ticket has a high cardinality: 681 distinct values High Cardinality

Dataset Insights

Cabin has a high cardinality: 147 distinct values High Cardinality
Embarked has constant length 1 Constant Length
Name has all distinct values Unique
Survived has 549 (61.62%) zeros Zeros
SibSp has 608 (68.24%) zeros Zeros
Parch has 678 (76.09%) zeros Zeros
  • 1
  • 2

The plot(df) shows the distribution of each column. For a categorical column, it shows the bar chart with blue color. For a numeric column, it shows the histgorm with gray color. Currently, the column type (i.e., categorical or numeric) is based on the column type in input dataframe. Hence, if some column types is wrongly identified, you could change its type on the dataframe. For example, by calling df[col] = df[col].astype(“object”) you could identify col as a categorical column.

From the output of plot(df), we know: 1. All Columns: there are 1 label column Survived and 11 feature columns, which are PassengerId, Pclass, Name, Sex, Age, SibSp, Parch, Ticket, Fare, Cabin, Embarked. 2. Categorical Columns: Survived, PassengerId, Pclass, Name, Sex, Ticket, Embarked. 3. Numeric Columns: Age, SibSp, Parch, Fare. 4. Missing Values: From the figure title, we can find there are 3 columns with missing values. I.e., Age (19.9%), Cabin (77.1%), Embarked(0.2%). 5. Label Balance: From the distribution of Survived, we are aware that the positive and negative training examples and not very balanced. There are 38% data with label Survived = 1.

[3]:
# identify 'PassengerId','Survived', 'Pclass' as categorical column and replot.
for col in ['Survived', 'Pclass']:
    train_df[col] = train_df[col].astype("object")
plot(train_df)
[3]:
DataPrep.EDA Report

Dataset Statistics

Number of Variables 12
Number of Rows 891
Missing Cells 866
Missing Cells (%) 8.1%
Duplicate Rows 0
Duplicate Rows (%) 0.0%
Total Size in Memory 361.6 KB
Average Row Size in Memory 415.6 B
Variable Types
  • Numerical: 5
  • Categorical: 7

Dataset Insights

PassengerId is uniformly distributed Uniform
Age has 177 (19.87%) missing values Missing
Cabin has 687 (77.1%) missing values Missing
SibSp is skewed Skewed
Parch is skewed Skewed
Fare is skewed Skewed
Name has a high cardinality: 891 distinct values High Cardinality
Ticket has a high cardinality: 681 distinct values High Cardinality
Cabin has a high cardinality: 147 distinct values High Cardinality
Survived has constant length 1 Constant Length

Dataset Insights

Pclass has constant length 1 Constant Length
Embarked has constant length 1 Constant Length
Name has all distinct values Unique
SibSp has 608 (68.24%) zeros Zeros
Parch has 678 (76.09%) zeros Zeros
  • 1
  • 2

5. Identify useful features

After we roungly know the data, next we want to understand how each feature is correlated to the label column. ### 5.1 Age, Cabin, and Embarked: features with missing values. We first take a look at features with missing values: age, cabin and embarked. To understand the missing value, we first call plot_missing(df) to see whether the missing values have any underlaying pattern.

[4]:
plot_missing(train_df)
[4]:
DataPrep.EDA Report

plot_missing(df) shows how missing values are distributed in the input data. From the output, we know that the missing value is uniformly distribution among records, and there is no underlaying pattern. Next, we decide how to handle the missing values: should we remove the feature, remove the rows contain missing values, or filling the missing values? We first analyze whether they are correlated to Survived. If they are correlated, then we may do not want to remove the feature. We analyze the correlation between two columns by calling plot(df, x, y).

[5]:
for feature in ['Age', 'Cabin', 'Embarked']:
    plot(train_df, feature, 'Survived').show()

From the output, we can find that: 1. The Age feature is correlated to Survived. Younger people is more likely to be survived. 2. The Embarked feature is correlated to survived. Passenger with Embarked = C is more likely to be survived. 3. The correlation of Cabin to Survived is unclear, since Cabin contains many missing values (77.1%) and many distinct values (147), hence each distinct value only contains a few useful tuples.

Hence, we could safely remove Cabin column. For Age and Embarked feature, we should fill the missing values.

Result: keep Age and Embarked and filling their missing values; remove Cabin.

5.3 PassengerId, Name, SibSp, Parch, Ticket: left features

We now analyze the left features.

PassengerId: it is just an id of each passenger, so we could drop it.

Name: there are many duplicates, and it looks not correlated to survival rate, we may drop it.

Ticket: it contains many duplicates and looks not correlated to survival rate, we may drop it.

SibSp: not sure whether it is correlated or not.

Parch: not sure whether it is correlated or not.

Hence, we justify whether SibSp and Parch are correlated to Survived.

[7]:
for feature in ['SibSp', 'Parch']:
    plot(train_df, feature, 'Survived').show()

From the output, we find that the plot is different for different Survived, hence they are correlated, and we will keep them as the feature.

Result: keep SibSp and Parch; remove PassengerId, Name and Ticket

5.4 Result

After the processing, we now left the following features that are useful to predicat Survived: Age, Embarked, Pclass, Sex, Fare, SibSp and Parch.

6. Identify Correlated Features

For now, we identified the useful features one by one, and removed the useless features. Altough each feature is useful to predict Survived, when we consider them together, we may not want correlated features. Hence, we first identity correlated features. This could be done by simply calling plot_correlation(df).

[8]:
plot_correlation(train_df)
[8]:

From the output, we know that: 1. The most correlated columns are Parch and SibSp, with a Pearson correlation 0.41. 2. There does not exist two columns that are highly correlated. Hence, we do not need to worry much about correlated features. However, as Parch and SibSp are slightly correlated in both computation and semantics, we may want to construct a new feature named Family, based on Parch and SibSp, which counts the total family members for each passienger.

Result: Construct a new feature Family based on Parch and SibSp.